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AflSIBACT 

Trends within the literature on program evaiuatlDn 
are revieiied and the iaplications of various trends for education are 
explored^ ulth parrticular relevance for distance education- Distance 
education is an alternative educational approach based on 
correspondence, broadcasting , and long distance communication between 
teicfcer and student in a non*home based study study situation* Iiio 
majoc trends are identified-'f irst # the trend toward the 
technologlsation of reason <i«e^# the acceptance of problem solutions 
by experts)* and second, a trend toward concern with preserving a 
huma-^ dimension (i*e., convivality) , Mlth regard to educatLonal 
evaluation, these trends have their counterparts in the^ move toward 
technical/quantitative assessment of performance and objectives and 
in the increasing interest in more individualized and responsive 
eval4iation methodology* A new model is proposed to counteract 
ihortcoeings of these types of evaluation models and of other models 
in widespread use today (engineering models, ecological models, 
democratic models, illuminative/responsive models, etc-)« This new 
model, based on evaluation as self *ref lection and convivality, comes 
closer than any of Jthe_ot^es_ao4el^i^^ 

— ^rt±aary^siTuiTi o n s ~w hen they try to evaluate without the aid of 
specialists* Specifically, the evaluation as self -reflection model 
itresses rationality, autonomy and responsibility, the community of 
interests among participants in a specific program, value 
differences^ explicit rules which govern procedures, and appropriate 
■ethods for evaluating the specific program or curriculum being 
studied* The .conclusion is that this self-reflective evaluation aodel 

* can help educators recognize one another's (and students) social 
being and develop a perspective which is more critical and reflective 
and less dominated by purely' technical solutions* (DB) 

• ,ae.productions supplied by BDBS are the best that can be made * 
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In. these remarks, I ' should like to address a very qeneral problem of - our ., 
culture as it comes to bear in the specific question before us today - 
the question of program evaluation in distance education. The quneral 
problem concerns the technologisation of our thought - our wiilinqness to 
subjugate our human, convivial intelligence and capacity lor critical , tliought 
to technologies of reason. In a world dominated by contending specialia 
arcane technologies and the proliferation of mass solutions to practical 
problems, we seem all too ready to give up our own critical intelligence 
and to accept as the solutions to our pwn problems the forms of life 
fashioned for us by others — where these "others" are "eKperts", 
'"specialists" or, as I should prefer ^to call them for our purposes here, 
**the technologists of reason", . ' • . 

Let me 4>et the stage by quoting the introduction to Stanley Aronowitz's 
(1977) paper "Mass Culture and the Eclipse of Reason i The Implications 
for Pedagogy"! ^ , ' ' " ' 

In his book The EalipSB of Reason^ Max FIorkhBime2''j foundar of 
the Fvankfurt' InatitutB for Sooial ResBOrah^ pi^ovided one of 
the most suQQinot foiwnulations of the problm engefidered by 
mass aulture. According to Hoi^khepnei^^ the signifiaayioe of 
the ahallenge posed by the massifiBd auLturs industry to 
aivilisation as such oonsisted in Us. assauit on the 
aapacity to engage in aritiaal thought as a meaningful foim^' 
of soaial disaourse. Horkheimer aared deeply about the 
content of critical t^^ught^ but with the rise of fasoiam he 
.became mopre Qonaerned with the speatre of the erid of reason 
'itself. In his view^ the aapacity of hman& to di'stanoe 
thmselves from the objedt in order to gain aritiQal pers^ 
pBQtive upon their soaial world adn no longer be taken for 
granted. The restriated language and thought aodes - 
- - \ -p^p^^^ ^-4)y-^he ^edu04i0n---^^^ to its teahniaat - - 

dimensions reaah far into the aulture^ enaompassing schools 
as well as conmuniaations ^ the publia as well as the private ^ 
spheres of disaourse. ... It is no longer a question of- whether 
ordimry disaoidrse is able to deal effqotively with issues of 
apeaifia ideologiaal and soaial content, '4a Jurgen Habermas ^ 
expressed it^ the new situation raises the issue of the 
aompetenae of people to effeotively aorrmuniaate ideatignal 
aontent. The issue is the -aapacity for theoremical or 
aonaeptual thought itself^ WJien people lack suah aompetenae^ , 
soaial action that trarsaends the strugglB for justice within 
the mpiriaally given rules of soaial organisabion and " 
disaovirse is imposs^ible, (p, 768) 
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The probiein to which 'Aronowitz addresses himself is that of our capacity 
- to think critically about our social world. Technology has outrun , the 
capacity of the ordinary person to understand; the legislative compItiKity 
and bureaucratisation of the administrative procedures of our soc,iety 
deny the layman access to understanding of access to broad arenas of 
social organisation* We have bacomei it seems, the instrumentB of the 
very social procedures and techniques we invented in- order to instru^ _ 
mentatise the world to our needs and our desires . ^ 

It is no solution to hope that these cultural trends can be reversed' 
only the romantic believe that the way out of our cultural predicament 
is backwards. Fundamentalist religious zest and "small is beautiful" ^ 
= tedhnologists uannot restore a presumed former state ot innocence. 
There is no way back into the Garden. The technology awaits its use ; 
the cultural desires which created it, once awakened, seek their ful-- 
filment. 

The scientists who developed nuclear technology during the Second World 
War faced personal crises because they knew^ that its capacity, once dis-^ 
covered, could not be covered up again (Jungk, 1960) . Our choices seam not 
to be about the usefulness of the technology (for better or for wor|e , it wa; 
designed to be useful) , they are about its wise use ^ we must control 
technology through wise and prudent use of its resources. To use it 
wisely^ we must develop new social, pplitical and proceaural resources 
for critique, the capacity to "out-think" our technQlogies and to see 
the limits of their utility. The "spaceship earth" image, coupled with 
a vigorous ecological movement^ has enabled us to see how our techriological 
development efforts have turned on us and against each otner. The "green 
revolution" liberates the capacity to feed the world's population but 
' creates monocultures^ cur economic systems do not permit equitable 
distribution of these food ■ resources withotit destroying the profitability 
which makes nverproduction possible. In order to comprehend and cope 
with these contradictions ^ we must develop;, ways of sharing knowledge 
which liberate che power of critique and ways of making decisions which 
allow us prudently to control their consequei ces . We require as muuh 
boldness in critique and the sharing of knowledge as is ^^senftly invested 
in the production of technologies and their use for profitable purposes. 

These are urgent imperatives. In education, as elsewhere, in society', we 
need to re-establish the means by which we can recognise one another -s 
social being and develop a critical perspective from which we can more 
reflectively .pontrol our development as a society and as a c\ilture, as 
well as our development as a political economy. 

The problem, as Aronowitz points out, is that we liave been so dominated 
by the technical problems of our technological age that we tend to be 
satisfied by purely technical solutions. Thefte tend to be "mass 
solutions"! ^abstracted, generalised, universalised , pragmatic strategies 
for solving probl^s of a given type or class. They may or may -not be 
solutions to our own problems. More importantly, because they are "off 
the shelf" solutions, they may distract our attentiori from the reality 
of our own situation, leading ,u^ to see this or that aspect with which 
a giveji' technology can cope, and ignoring the wholenesLi , , the manifoia" 
reality of our own situation. 

There is a'place in War and P&aae where Tolstoy is discussing the ^ 
^question of free wil4 and deteminism/ He makes a point there which 
* bears clo^eli^^ on our discpssioni \ 



thqnke tc the ^ncst vo'^^vfuZ i^^^^pcn :^:hi/ihAK.\ — ^=^' 
J-:ffuMt^n of zvinted matter ^ — VfU L/u^c^t-ij'K :/:,; 

In distance education, we are, of course very much invol%'ed in thu 
business of the diffusion of urinted matter (Tolstoy wouli unJoubtt'dlv 
have said the same for our audio and videotapes, and the inireniiin.i 
camputer revolution in distance aducation) and so th^sre is. a ! arallel 
between our primary t^k and the evaluation of thut task. Ou thi:- o!iCi 
hand, there is the trend to mass culture and the tacluiulonisation of.. 
rMson; on the other, a concern fat conviviality, f-r ^ L-^jJcrvin.: a v^ry 
human dimansion in our affairs. In distance education there has lonq beer 
a concern for humanising the processes of lonci^distance communicat ion , for 
preaervinq the intimacy of the relationship between che i^ariicr anl tiie 
teacher/ At its worst, this has been regarded as a purely technical 
matter: a matter of motivation only^ n&fi,Af reaard for the hiiman diqnity 
of both actors in the 'educational ericounter. . In contrast to this technical 
view, Ivan Iliich (1^71) proposed a system of "learninq weo?3" lor ri^Lworks , 
or e^ichanges) whose aim was to preserve conviviality whil^- al^o maintaining 
the possibility of education ^at a distance. These werei reference 
services to educational objects^ skill exchanges, peor^-matchinq , and 
reference services to eduGators^at^large . There was considerable interest 
in those revoiutionary ideas in the early '70s, but they seem to have 
succumbed to the power of another image — the mass culture imaqe — of 
what edacation is and should be. They are now frankly dismissed by many 
as hopalessly romantic and in any case as more appr^opriate to Third World 
educational problems, not our own. 

Perhaps the mere mention of that name — Ivan Illich — is irritating. 
Surely # SDme will sa^, those disquieting images and disturbing possibilities 
were laid to rest six or seven years ago. Or were they? It is my belief 
that the CQuntervailing tendencies to technologisation on the one hand 
and conviviality on the .other are as strong today as in the late l960*s and 
the early 1970 's. As much as we may be concerned about the systems wejhave 
developed, we are concerned that we can recognise ourselves and one another 
in them as persons, hot merely as instruments of economic processes. 

This contrast between technologisation and conviviality has its counterpart 

in educational evaluation. On the one hand, there are trends abroad for 

efficiency in the management of mass education systems which represent the 

tachnologising view in educational evaluation. The development of a 

program for national assessmant of educational performance in Australia 

is a stark manifestation of that trend. On the other hand, recent 

developments in evaluation methodology like Stake's (1975) "responsive" 

evaluation and Parlett and Hamilton's (1976) "illuminative" evaluation^ 

express an interest in conviviality, manifested in their use of the 

VGrstehen method which aims at empathetic understanding. They invite the^ 

more widespread use of interpretative methods (as opposed to the empirical^ 

"analytic methods of the "engineering" or "agriculture-botany" approaches)*^ 

*» 

The literature of •'"program evaluation" (though perhaps many do not now^ 
grace their present evaluation efforts with that graceless name) reveals 
that those two tendencies are chafing against one another within the field 
and in the work of program evaluation. Many may think of program 
eyaluafciop. as pure tech^logy and as technologis^d . Many ^writers in the 
field are guiity, I bemevev of prpmoting a technologisation of our reason 
in* advocating models oi evaluatimi based simply on inadequate models of ' 



raM .nal i4nageinent.. Liberal and qentle souls thrust Luto ..r.;..n u.at unu.. 
evaluation settings they choose the carapace of rational manaq.noc.t a^ 
orotacxion. Spare a thought for evaluators: their judqement^ rual 
Deople vulnerable and have real consequences for organisati ma . .t^--. 
'sometimes an uncomfortable role (thouqh a ^ood deal more c, H-,: ortaoic 
than ^he role of the evaluator) . Some evaluators find it saf.r u. c^..o.k. 
the' dispassionate, objectivi.t methodology of the "tional management 
approach to blunt the knife-edge of conscience. Qthers-bel^wo ^h., 
pissibility of a purely rational society in which utilitarian Wtn.w. .an 
ultimately advantage all. 

Still other evaluators are willing to find more human, solutions arid to try 
tne hLuman wav: they see conviviality as a qoal , even in conflict , and 
propose less^ rationalistic methods. They demonstrate that, conviviality m 
program evaluation is possible. , ^ 

I hope to s.how ■that.it is possible in program evaluation to icef. ilve some- 
of these issues. Technologisation of reason in the name of objectivity is 
not enough- conviviality by itsel^ can be purely romantic notion. Beyond 
conviviality is the flintier, c-immer prospect of social criticism- 
"flintier" and "griiraner" not because the prospect of social criticism is 
grey, but be«ause the task is a /narder and more demanding one. It is the 
task of rebuilding institutions on the shifting sands of political alliances 
and defining their images in the irdBta of ideology, of contending -rhatorics . 
and aspirations. As such, it entails not only organisational change but 
cognitive change. As the Chilean physiologist Humbertd Maturano (1974) has 
argued, any attempt to change society requires us to change the way we 
live; and this requires changes, in both- the cognitive and cultural domains. 

Before proceeding further, it may be as well to make one important note 
with respect to program evaluation in distance" education, i^hilo m many 
respect's distance education differs markedly from conventional education, 
in questions of program evaluation, it is generally similar to conventional 
education. To be sure,' there are differences in the character of the 
programs to be evaluated, especially in relation to the "delivery systems" 
* and modes of communication with students, "but there are nevertheless strong 
similarities I distahce education course developers work together as 
conventional course developers do, the institutions offering distance 
education are educational institutions, and so on. In what follows, I 
have chosen to speak about program evaluation in general,- 1 believe that ■ 
my arguments apply as much to program evaluation in distance education 
as to more "conventional" forms of educational provision. 

PROGRAM EVALUATION ■ C 

It is by now more or leas customary in evaluation to diBtinguish between 
evaluation and assessment . Assessment of student performance — evincing 
a concern f the outcomes of a course, curriculum . or whole program ~ is 
only a small part of evaluation. Beyond student assessment, I would like 
to direct your attention to "at least .the evaluation of student learning 
(tto evaluation of- the opportunities for learning provided by a courDe, ^ 
curriculum or program), curriculum evaluation -(the evaluation of the whole 
set of educational, social administrative and physical arrangements mad^a-,^ 
by teachers and course developers for students) , and program evaluatio,p \ . 
(which includes the former typfs >of . evaluation but expands them into the ? 
realm of the whole institution ■providing these- educational arrangements). 



To pu.t it very crudely indeed^ evaluation might be considered at ioar luvul^i 
each including subsequent levels within its purvie\v*^ (1) pt-c* iram "evaUiat ion * 
concerning general institutional arrangements; (2) curriculun ovAiuation, 
concerning the educational arYangemonts of whole curticula and rAirticular 
courses; (3) the evaluation of student learning^ cdncerninq 'i-hu anpor tanit 
' for learning created bv a particular teaching/learnin^J encouiitkfrf and (4) 
^ student assessment, concerning the outcpmes pt student l-Garn^aiu . 

, At the most general level, there is a range of alternative mcsdels of 
^^valuation. The relationships between levels in these models; are blurred 
and of ten -obscure , howeveriit would appear, chat most authwirt; in t'he field 
have bean guided in their model^building activities by an ovtn^^ riding imaqe 
of the activity of e%^aluation. For example ^ Kiiox (1972) rias in miiid a.n 
image of the director of an adult basic education program whci wants to 
collect fairly comprehensive data about how his prbgram works; ^ who it 
serves^ how they experience it and where it can be imp* roved , He provides 
a list of factors to consider under eath of seven broad sub^headinqs for 
an evaluation ^plan • 

I. Background- regarding the anticipated contribti "ion of 
evaluation to program improvement 

A, primary purpose of evaluation for this course 

B. Symptoms that^ indicated that the eKisting proceAires 
. ' ^ were ' inadequate 

II. The approach to the evaluation plan 

, A. Ideas about evaluation to keep in mind , 

B. Types of evaluation data to be collected 

C. Functions of the division that may relate to course 
effectiveness 

D. Probable^ methods of data collection and analysis 
Using the results. . . . . 

Stake (1967) ^ seems^to have in mind the inquiring teacher or outsider 
observing a teaching/learning encounter^ or a sequence of such encounters 
held together in the framework of a course. His "countenance" of 
educational evaluation consists of a matrix of antecedents, transactions ^ 
and outcomes by intents , 'observations , standards and judgements; the whole 
being considered against the background the rationale for the program* 



Intended Observed Values Judgement's 




Antecedents 
Transactions 

OutGomes 




Rationale 



Descriptive Data 



Judgement Data 



FIGURE \\ The matrix of Stake *s "countenance" approach. 



Stufflebeam, Qt al. (1971) Wave a model in mind of the rational manager of 
an input-throughput-dutput system which leads them to set out a range of 
considerations ur^er- each *^af the global headings of Context # Input/ Process 
and Output (hence CIPp) which can guide the evaluator towards data likely to 
' be relevant Jin making decisions about the whole program, y 



; TKes#"exampjjys S"eTve only to iilustrate the themes that tho nvulu^L^^u 

'whole program seems to be an extremeiy' comulex business ^ uiui that, in ordyr 
to reduce this complexity to manageable proportions . special istn in tlio fi^ld 
have been guided by images of the task of the evaluatoi : harj^iud' nrw'^jr.Lin 
manager (in Knox*s case) , o bserver, of educational encounters »irriHakG'ri} , 
I or rational manager (or %hB servant of a rational manager, in Stii f f lui-uam ' a) , 

For those interested in program .evaluation in distance education ^ anLl 
especially interested ^rt* the eva'luation of a distance education r^roqram as 
a delivery system^ any. of these three models might suffice. But/ as my 
introductory remarks suggested, I think there are considerable, dangers in 
treating an educationaT program as no more' than a delivery system. By 
doing sQ , we may fall into the trap of technological reason^ the traT? of 
treating the delivery system a technology which may be fine-- tuned to 
imp^rove its functioning or its efficiency, and thus ot failing to subject 
" ^ it to substantive criticism as a form of life for students, teachers,* 

ancillary ^staf f and observers. 

I would like to turn now to a consideration of five classes of evaluation 
approaches or ^ as I should prefer to call them, five images of evaluation 
which have some currency today. These ^are the engineering model, the 
organisational model , the ecological model , the illuminative/ responsive 
model, and the democratic model. After considering each briefly, I would 
like to raise one or two general matters and thenuiffer a sixth model, the 
.model pf evaluation as self-reflection , which I believe surmounts some of 
the difficulties of the other five. The sixth model sets out to incorporate 
the two notions discussed earlier^ conviviality and critical reflection. As 
it happens, the model of evaluation ^as self ^reflection is (I believe) also 
the model closest to what we do in ordinary situations when we try to 
evaluate an educational prograin without the aid of . s^eciaiist approaches 
to the evalMation task. ' " " - ' , 

THE ENGINEERING MODEL ''^ 

The engioeering model of educational evaluation is the one most^ fojniliar to 
curriculum and instructional developers. Writing about this model ^In 1979, 
David rd^Jjkins ^ Barry MacDonaldi.; Gajendra Verma and I had this to say^ 

TTaditionally^ eduaatianal evaluation has attempted to provide 
aurriauiim- buil4&^*^ with, some ohsak on how well their intentionB 
have been fulfitted in tPie eduQational pBrformay%ce of the pro- 
grcmne undei" development. This has resulted -(in g&nQval) in a 
" ' aonaentration upon dev&topeT^s ' instruationr^.l objectives and 

. - related studemt lemming outaomes. This traditional model 

(whiah we aatl the ■engineerifig model' sinae^ it repPesents a . 
" teahfialogiaal solution to the problem of aurriaulum development) 
was fir*st artiaulated by Tyler in 1934 and subsequently ^ : 
published as a aurrioulim/ for students of otdrriaulwn-. in 1949. 
In its most hfidely rscDgnised form^ it entails the 'foltomng 
' 's . stages: ' " . ' 

i. Seaure agremerit on the aims of the aia^rieulim and/or. course 
* ' ! of study; 

I' * ,2. Express these aims as -objeatives- (i.e. empliait by ^stated ^ 

student b&havioia'g^tfuit the aurriaulim is intended to 
produoej j f j " * • 

-3. Devise* and provide ewperMnae that sems likely to enable 
* / * .the tewrners to behave in the disired.wajy; 

^ ' ■■ . 4. Aas&BB ^he aongrumae of Btud&ni^mrfom anae a? td obdeative^ 

O ^ ' ' = ' 5, Vary ^ the 'trgaPnent' until b^mviaur matahee objeatives, ' - 

ERIC * - * — — 



pro<jt.tS3€8^ has exertt/d a pawerfuL in;l:iC}:^c .^k buvi^L^ulii:- 

=-- —emZ uatia n- jn Jjijr/i. aiJ^g tjf tha .A^la>:i li^. dLailna^lhc^lLdl 

thjentii ye^rs. It has 3ubBta?iHa2' m^tit^ ^.f ^ {rn^d. ^ ' ^« , 

ratio^u:rl:tlf^ uHlitu LVid praatiaalitti . ' Iz 3v^^Szyui^ th: ! ■ 
^ ^ /taak the^^evaluatoi^ that of rneaaurty-(^ Trie v^t^^;r :v 
uhzufi tritj p^rfamaKa£*af students c:j^;oocd c.^ a • ^u^^zr 

eduaatia^iat ti'eatm^nt corresponds wick the devt/lcr:, r 'i' 
expressed intartions . Ayncst all suBsamen^ c^'^ahiatiOK ^ 
theorists have stm^tm n*om a aon3ideratia>T c>^j thiiS ncd^l^ 
Qithm* m:t€ridi>ig ami r^jinim it^ or r^aazinr? anrzi^^^z 
■ Cpp, 111-112) ^ - ^ ' . - ■ / 

In a critique of the engineering model, we marshalled criticisms which, 
were becoming widespread and grouped them under s -jral cateqories: 

(a) Problems of sufficiency' ^- . * ' * 

The engineering model is too narrow in its focus. Data about student 
outcomes are insufficient for making judgements about programs. Data 
a_baut program circumstances and processes are 'as 'imp>prtant as data about 
outcomes in reaching judgements of the overall worth of a program. More^ 
over, as Stake (1967) most cogently argued, a variety of values are rele^ 
vant ia reaching a judgement of its merit (not just those of the program 
developers) , and judgements will vary with the valCWr of the judge. His 
"countenance" paper argued that the plurality of perspectives and 
judgements needed to be taken into account. in . evaluation . 

(b) Problems of^ specification ^ ^ . . ' * 

The behavioural specification of objectives required by the enginueting 
model also proved unacceptable to evaluators who attempted ^o use it and 
go beyond it\ Eisner (196-7, 1969> proposed that "expressive objectivaa" 
should also be considered in curriculum, not just behavioural. Moreover, ' 
it became clear that teachers did not always fol4ow developers' objectives, 
that they pursued many different objectives simultaneous'ly in their, 
teaching, and that different teachers pursue different' objectives . ^ ^ 

.Behavioural speGif ication, while apparently acceptable in a closed world 
of the developers' objectives, did not meet the requirements of the real 
world of education, Scriven (1973)' also mounted an attack on the 
objectives-based model, arguing instead for "goal^free" evaluation, in 
which the evaluator made a deliberate effort to avoid learning the 
program's goals and thus to esoape the "tunnel vision" of looking only 
for the intended effects of the program (n^ distinct from unintended or , 
side-effects of program operation). Finally, th^re is the problem that 
the spfcified objectives of the engineering model depend upon consensus 

.about what is to be learnedi in educational programs where broader social^, 
aims are pursued, such consensus cannot be assumed. Behavioural objectives 
may propose a consensus, but curriculum developers and evaluators cannot 
assume that specification will reduce division a^ut social goals; on the 
contrary, it may increase it, ^ - _ 

(c) Problems of measurement 

The engineering model requires ^that criterion behaviours for a program can 
be specified; it also requires that the discrepancy -between desired and 
actual achievgment can be measured/ But |tKere are massive technical 
problems of developing tests^ of adequate validity 'and reliability for the 
program, itself and the population to be tested. Walker and Schaffarzick 
(1970) demonstrated how often inappropriate tests have been used in 



comuar Lsiona between innovative curricula and "converitiunal " al tonKir ivt?^ , 
and how often the measured=' differences reflected tlfe biasiis or tlie. teiit:-i 
rather- than the qompetences of jtudents/ Fur thernajru , .in areas wht>j%^ 
■^brHaa^^atti tudinal "o^ pursued- by a~i^rG<rram.,- x^it^i _ 

are notorrously ^ubjac#"to bias — and . the biases of the *testa mav^iL^ may 
■not be the saine as the biasea of the programs under Mnsiderati^. 

"(d) Problems of explanation 

•As Hastings (1%66) / Stake (l567) and Parlett'and Hainilton tl97eT , arnonq 
, others have pointed out, the engineering model, with Lts emphasis on 
outcome measures, fails to give evaluators or curriculum lievelor-^ers^ii- 
understanding of why a program fails when it fails/ It ^/ails to show how 
learners leBrn, or why they learn some things and not otfiers . / Most 
audiences of evaluation reports are^ interested in the condit ions of . 
learning, not just what is learned. Hence, there has been a orowth di ^ 
studies of educational "transactions" (Rippey, 1973),,. the "social 
anthropology" q€ educationai innovation (Parl^tt and Hamirton, 1911^ i 
Smith and Pohland, 1974), and the conditions under which learning takui? 
. place (Kemmis, 1977). i have addressed the problem more generally in 
proposing the wi^er use of "idiographic"'methods in eyaluation (Kemmis,. 
1978) in which close descriptions are made of the learning' prpc&s^s in ,the ^ 
evaluation of student reaming. ' ' ^ ^ „ , 

• (a) Problems of epistemology * ^ * . - 

The finals most general S'jt of objections to the ^.engineering model relates 
to its basis in a positivistic view 'of knowledge and learning/ As we shall 
see later in this paper^ this '"objectivist" view of ^knowledge turns out to ^ 
be narrdwly sciantistic (see^ Habermas , 1972)., that is it is premissed on a 
view of science that regards th© essential ^pjoblem at the nature of truth 
as solved." according to this view? only technical problems in science 
remain; tfee natur-e of science itself is taken to be unproblematic . The 
danger is, that the engineering ' model appears to treat evaluation data a^ 
value^f^ee ("objective") . Yet, as Clarence Karier suggests, evaluation, 
always has an ideological component it always occurs Hi thin a value- 
framework i ^ " 

' It is my understanding tPiat evaluation is a'^aomplex pi^aacss of 
assigning values td' phenomena^ ukile idtsaiogy represmits timf 

^ set of valums and attitudes whiah go to make up the^ aompo^ite 
via-6ur£ of the saaial and individuals philosophy by whiah n&n i>r 
'a given aulture pi^ofess to live. In this aontext^ evcduation 

- inevitably aaaurs within some kiml of^^v&tue orientation as part' 
"N^ ;^ of ah ideologiaat frmnewoi^k. ^ . . 



To some observers, the whole corpus of scientific knowledge as we know it is 
a uniquely, evSn'peculiarly Western product, reflecting a faith that all 
problems — of , production, of action, e-ven of the soul — will yirld to 
sciehtific investigation. Only recently- have there be^^n indicatiofis that ^ 
this faith has been substantially dented (for Example,' in the failurop of - 
American military science and social engineering to win the war in .Vietnam) . 
But these dents. show little sign of breaking that faiths (as David Dawkins ' 
paper at this seminar so ably demonstrates) . ^ ' , ^ ^ 



These problems with the engineering model might seem to have left it reeling, 
unable to attract champions in program evaluation today. But it is not so.. 
The image which the engineering model mbodies is a powerful, one for our 
technological society, too powerful, perhaps, to^ be easily recoverea from 
the store of our cultural archetypes and symbols, and subjected to critical 
scrutihy * ' ^ : \ . 



. Erauty MacKenzle ^nd "Papps (1975) have cummented on chti.- "rnvth.. ^ 'ly " 
, surrounding'' aduo-afionaL, development and its^' Gvaluation . Ainoiiq tiic- , 
my-t ho l-ggxcaX- as pec t s_ they i^g p Y 3^re the .tolluwiiiq ^ that ^^1 u-.irniiiM la 

• to liberate the ieartier (oo tho contrdry^tney~XKf1lTt^'rrr't^^ irt!-^^i n. ^ .... 

* aiithoriraridn mocf^l d^teo'chiiiu and learninq embodied m tho jr.iKd.ji d 
view dOf ^CicationaL development), tha^inyth that stiutunt^ aru l:u.; 

—-(in their view teachers turrt out to be tihe clients) , tha myr.h thdc .t-rib 

iTiiiln j^.iu^^"*-: d' cc?'^? of^ toe paradigm' i'S the .improvement » )f istudunt^/* ' ' 

learnina efficiencs' (it turns out that students must adiust to tiit* 
........ ^ ^ . . . ^ . . . 

materials, not the other way ai^ouud) / the myth that ytuduMt a.chic^'/uTi'oiit 

is the all^imDOrtant aim, (it turns out that vl.sible niatez^ iril,^ , -lu^f * ^ 

student achievement is -what sounts) , the myth that knn'wlqd-iu ] li _^'oi"uitrifctt_-a • 

like building^ with blocks ( the irj observations led thenKto see thu? rrc>wtii of * 

icnowledge as Ehe student's reconstruction oi a whcTle uverlaiidiKj sul ui 

olusters of concepts and n\athods ^which. constituted thq di^cirilinu) / and the 

my t h that i nd ividualisa t ion of learn i ng ' im : i \''ed l^oa r^i ng ach i e%' ^.^m^ c ; \ t h y 

qivinq ^the learner control ovel^the learriing process' rtheir ob^iervatLon ^ 

'was that l.earners; had con t col onLy over the pace tind palace of leariitnuj in 

^ the tertiary course ^hey stuSied^ students demanded a more found kirid of 

control over their own learning — ■ control over" th?^' coqnitiva or u an Ib at ion 

of the qourse) . ^ ^ " 

^These mythological elements of the technological approach to educaticmal * 
deyelopment identified by E^aut/ MacK.enz ie^and Papps suggest that. the 
Image of the engineering 'model has pressed deep into oar understanding of 
educational development, but , that *the. image is misleading. This is why it- 
has a "mythological" Equality* For this reason it may be asserted that- the 
en^iheering model -.of adueational' development evaluatidn ^repres^nts a 
^"teclinolo^isation of reason" i it limits our thinking abou.t, these ma,tters i 
Uo technical* and procedural' dimensions ^, This reduation is fundamentab to 
tHe model. It routinises the quintessentially kuman' process of education. * 
' so that it becomes more subject to technical managenient. In the next model' 
to be discussed, the orgaTiisation^ model, €his -routinisaticn is prdje^cted ^ 
onto an instdtut-ionaL^ plane: the ,int#rests^-.Qf .efficient management* are 
extended to the management of institutions, as social systems, 'subject to 
analogous technical thinking . , ' ^ , . ^ 

■i r 

X have devoted so much space to the engineering model prajriarily because it 
'is so profoujid a part of our^culture, expressing so easily the technolo- 
gisa^ion of reason* It is, perhaps, the hardest image of program evaluatioh 
to extirpate from our thinking. Many evaluators, have learned its limitations, 
however, simply by binding themselves to its dicta tjBS and 'then by considering 
how little o,f the life of ^ the- program they have captured in therir methodo- . ^ 
' logical nets by applying it rigorously. There are* others in the world of 
= professional "ev^luatian,, though, who see things ^differently . The task t.o 
them "IS simpler; the world beyond the technology c^ore dangerous. With a ^elf^ 

* assurance rooted dee^s in our technological culture they model" ^hemseiveB^pn^T^" 
Ulysses,- stopping the ears of their cbmpanions with wax and tying' themselves 
\q the mast^of their, methodologies , Thus' they avoid, they think, the gra^p ^ 
of the sweet sirens of the wider world singipg from the shore, ^ ' , 

. . ; ' ' ^ \ 

' THE ORGANlSJkTIONAL MODEL ' ^ \ 

^ \' 

* " . _ . . , ^ , \. 
The. second image of program qvaluation is based on the imaqd of the program \\ 

as an organi^tioh* It is represented by the view of Knox (1972) discussed V 

* earlier (the harried program administrator attempting to gather Sata on i\ \ 

the wide variety of concerns which might bear on a program) * This moddl :\ 

has become widely, established in procedures for school accreditation, \ 

" school self-study^ and schotfl review. The procedures have been further • | 

developeca and refined in the context of eya^uation in higher education, '| 



Paul DrukSHeL'j (1076) HZ'L::^.^^' vl j.:: " \* Ki'u!:u : : .'k itr rut iw^u>iit itivi' • 
of the q^nre . ' * ^ * ^ ' 

The aim on the oraanisat ional, modol is ossantiaily bureaucrat ic j to i^orvs? 
proqrarn manaqers (deuid iori^^kers) with the ranqti oi : )rniu l i ■ 'ii iul^'V.lnt 
iji^"lcgef5lng rheir [rrt:>^ram3-- i^.-iristitutiDns on the rLih*_ trutck, 'rhtj model 
is one of rational manaqament, tiia image supporcud uultcri\il - v.j lij^n^ {-t^--.-.,.. ._. 
scientific rationality., ecii^nQmic etfici^^ncy, and consensus about an ' * 

institutional missiqn: in short, the image ij of the orqanisation maru 

There is somethiriq peculiarly lulling about the organiBationai model ^ ay 
each dimension of the institution' covered by the model is mer;t:ioned, a 
litany of relevant information Is recited. Rational maiiaaement.- Lurnrf ^>ui 
to be a daunting task.* So many things to be taken into account by the wi^^e 
and prudent avaluator! Thefe is not space^ here to recite the list in full" 
^ut it may be helpful to •intone at Ifeast the, macro-headinqs /' the main, 
dimensions, listed by Dressel in his considaration of institut j.onal self- #1 
study (itself ^but one weapon in the arsenal of the iristitutional uyalu^ltor) 1 

4* ' Determining jyistitution viwpa3e& ^ joal8^wid eduaatioKal objeatives 

triB ds-gres' to whiah_^in&titut.iorial uubion ts are in au^ci\i uith the^^ 
jaul&^ i^esoiwa^M pfdvisionj and the dij^jvec tc whioh t}w -ipdla are 
^ aaknawl&dgdd by n^ndividuuLs and subunitc tin'ough^ut (iJik:: 
institution) . \ ^ ^ _ - . ' 

B, MBasuving 'eduaationml mid othex outaom^ (inaluding issufjs 
- of wkat t}i3 olitaomes cn'e and wKethen all i>i the institutidii 
pio'Sus thmis the appi^opriateness of th^ outoomaB to the 
ali^entelBj trw methods af asseBBment €mpioyed^^i7isti-tutiarial 
effec-tivefWQS in aa/iieuiKy othei^ goals like uor^tfruyiitK sei-vi 



, Evaluating learning mcpei^ien&es in terms of the desired ^ o utaomeB 
Inaluding is^sues of the types of ^perienaes provided for^ 
Bttidents and' the extent of their imoZvement^ in them^ Judge- 
men ts_^ oj all I'elevmt gi'^outys '~ teaahers^ students ^ adnini^ 
stratars — dnd assessment' of institiitional alimate) ^ 

_ b* Evaluating the 'ade0Moy_ and,^tilisation of resouraes in terms 
= of deBii*ed. outaomes (inalUdina instinj^a^ional staffs learning r 
resouraes^ finanaial resouraes] and physiaal plant ami 
equipment) ^ , • ■ ^ " 

* Evaluatinfj the planning and deaision-mhking pvaa&sse^ in^ terms 

. i of the desired outaame (inaluding questions of who is involved 

in these proae&se's^ the groups beyond the iristitution who 
' > ir^luenae thm^ student involvement^^ and aonBideratians of ^ 
0Vidcnce Related to^aatherirw this data) ^ ' 

- t\ Inter%iretimf .objeativeij means of attainment^ and Bvidanae^of 
♦ g-^^^i^&^i^L^^P new faau%ty^ to students ami to ihe^vublia , 

There is a tretnendous Jpuriousness cir^l * misplaced earnestness abou t'" s'lrch > l;i&fes-— 
of relevant factors* One cannot their iTi-prinoiplo ralevance to 
Secisions i about an insstitution^ the notion- tflat any evaluation can 

achiaire all of the data-gathering,' analytic and reporting tasks^ se^t out is 
^bsurdr 'At* best, the' e^^ercise is, superficial, and misleading, at^worst/ it 
IS a machani^. for ^subordinating those i nvolyed ^o institutional goals 
whitW aye .expressed at a supra-individual,^ rhotorical level. The goal 
of "the examined life" may vbe worthwhile, but ^he.rV'-fti^s't be« time left t® ^ • 
ifve it as well . Following the co'mprtehcnsive . prescr ip^^n of an ovaluation. 



ft 



, system such as this is & sulre way to get the inhabitants of the institution 
, to lea^e evaluation -to the,- experts . There is- enough experience with such 
schamas in Australian schools for us to iiave some certainty that tiiey tend 
to'disaffect staffs to involve them in extensive data-gathering eKt5rcises 
which eat Up scarce resources of time and money/ and to yield little in 
tann% of actual change, ' ' * 

It if worth noting r too, that the organisational model begins as the 
engineering model. does i with a prescription of goals. All else hanys on 
thii prescription* The same objections to goal-oriented evaluation apply '-. ■ 
here as to the engiheering model* . ^' 

; . Put simply, there' are fouf kinds of problems with the organisational models 
' it takes a view of the institutional wor^d as a complex set of interactiTig 
^variables and is driven to absurd lengths ^o control this- complexity i it is 
so comprehensive as to defy genuine applica'^ion^ it is politically insidioua 
— it Subjugates the critical perspective of'^inhabitants of the institution 
to the perspective of the institution as a co^orate eiYtity in its own right i 
ahd it is goal -based, and therefore open to some of the'^fcriticism made of 
the engineering model* , . ^ " 

The .^rganisa'tional model I have presented here, the institutional self- 
study, has the redeeming features that it proposes -the engagement of 
participants in a program in its evaluation, and that it suggests the 
enormous variety- of places where something can go. wrong with an organisation. 
The rfirst is essential to good program evaluation, the second may be ^helpful. 

' But as\l have t^i^eS to show, too, great an earnestness about follpwing the 
prescri^ti^>hs of the .model will lead those involved away from the issues 

. 'they can^,d6 something abouf,'^;. In .ihif sense, it exemplifies clearly the 
^ danger of the technologisation af reasom it may deaden critical awareFness^ 

- ^in' the . sidw grigd- of its progress rather than heighten self-'Criticism in 
^ convivial debate^ ^ . . . . . 

/ taE ECOL^ICAL' MODEL ^ ' \ 

y * y ' ^ ' ^ . ^ " - 

r ^ A new model for educational evaluation is merging out of the ecplogical 
\. psychological litera.tufa (e.g. Moos, 1976KProshansky, 19^) .-A recent ^ 
' t paper by Urie^ Brpnf enbrenner "The exper^M of education" (1976), 

points thfr way f or -:'a series of— davelopments which may be quite interesting 
' ' ih program evaluation* Bronfenbreriner argues that „ programs are only 

rnadequately tested ih artificial' or laboratory jottings i that- when And. i 
1i6w pteopla;.lea2f is affected by the relationships between learners and the 
surroUhdlngs in which they live and' the relationships between the settings 
whicS^;c6ns^itute their , linvironment; and that the; methodology of choice must 
. be wWat he calls "the ecological experiment". He employs the terminology 
^ ' (adapted' fr dm: Brim , 1975) of. mrfaro-'Bectings (the immediate: settihg containing 
, the learner;-— e^g* *home day-care centre) , meso^syst ems .^("the interrelations 

between the^ major settings containing t.hs learner at a particular point in - 
% his life' —; , home^ jcHool, pear-group, television) , ^ exo-systems (an 

extensioh of the meso^system embracing coricreta, formal 'and informal social ' 

- structujfeS = — including, the major insti'tutions affiecting the life of the 

\ child, jlike neighbourhood, "mass media, agencies of government and informal 
social networks) , and macro-systems ' (the overarchdng institutions of the ^ 
culture or subculture — like aconomic, social, pqMtical and legal • 

- systems) . - ^ ' ' 

, . The image which urlderlies this tobdel of program evaiuation, if such it may " . 
be called,= is" the image of the evaiuator^ as ecologist, the program as 
^ecology. I. mention it here because the image is a powerful one for us 
^ today, '^e modal expresses a widespread desire in =ou.r culture to ^ understand 



our. social life not j'ust as .a set of interacting variables, but as, organic, 
structured and functioning. It sees a. program as "alive", but more than 
that, as part of a "living" context, 

■Bronfehbrenner.'s paper makes interesting" reading , The early 'results he • 
cites in support of his approach £ire-j encouraging . Iz will be,, interesting 
to see whether thes-e complex ideas are taken up widely and tttsted 
rigorously. 

But these are not the methods for a program evaluator. They are methods . 
for a social science researcher who has the leisure and the .distance from 
a social reality to treat it as ah object. In, detail, the" approach falls j 
prey to the dangers of positivism cited in relation to the enginearing 
model, and the problem of compiexity raised wLth respect to the organisa- 
tional model. (I do nttt mean that, complexity by itself is a bad thing; 
rather, I main to suggest that the kind of complexity these models deal m 
is hostile to understanding because it fragments our view of the program 
along the J.ines of its own analysis, not according to the ptoblems 
perceived and experienced by those 'who inhabit programs as "worlds" m 
which they live).' . 

These first three models mm engineering model, the organisational model 
and the ecological model) share a critical feature. All reveal an 
"objectivist" episteinologyi the evaluator attempts to study the program 
"objectively" and dispassionately. All share an interest m value- 
neutrality with respect to the program. And all display w|at Habermas 
C1972) calls a "technioal" knowledge-constitutive interest . ^ 

These approaches manifest mbst caearly the technoxugisation of reason, " 
making critical Belf-understanding subordinate to program. goals, bureaucratic 
organisational imperatives , or the "life" of, the proqram as understood from , 
a non-participatory (non-empathetic). perspective. The next three models to. 
■be discussed eschew the objectivist stance ir^: f ayour of more empathetic and 
convivial approaches to Cinders tanding . - , ■ 

THS ILLUMINATIVE/RESPONSIVE MODEL . . • . ; ■ 

Recent evaluation literature has paid much attention to two .relatively new 
models for e^^luation- Parlett and Hamilton ■ s (1976) "illuminative" approach 
and Stake's (1975) "responsive" approach. ^ 

Stake's responsive approach attempts to address issues raised by program 
participants rather than to...import the ejuestions of the- evaluator into the 
program setting^ He states: . . , ' 

An sduoational evaluationHa resvonBive evaluation (1) if it 
■ ■ ■ orients mope dimatly td^progrm aativities than to. pi-qgrarn 
.intents, (2) if it reepor^B to dudienae requirments for 
' infomation, oM (3) if the differmtmlue-pempeat^^^^ 
are referred to. in reportim the suoaesa and fa-Llure oj^tho 
progrmi. In these three aoparate ways' an evatuation plan aan 
be reBponsive. 



2 Knowledge-constitutive interests are the human interests 'that motivate 
^and guide the quest for knowledge. . The empirical-analytic sciences . 
(eg" the physical sciences) are guided, Habermas argues, by "technical ■ 
interests that is, interests in technical control .of the phenomena studied; 
• the hermeneutic or interpretative sciences (e.g.. history) are based ; on a 
■ "practical" interest, that is, in understanding problems whic^ave arisen 
and in accumulating experience which may guide us to act more wisely in 

i ,future;' and critical, social science (e.g. political economy, some 
, approaches in sociology) is guided by an 'temancipatory " interest, an interest 
■■■ ' in emancipating people from the mystification of ideology, the dictates of,.. 



habit and precedent, or the compulsions of self-interested authorities. 
In deciding how to address the. progr^/ Stake *s responsive evaluator is 
guided by th© issues' which concern those in and around the liroyx'^ani. Thoao 
issues guida the quest for data and the writing of reports. The evaLuator 
is likely to choose to make portrayals of the pz-ogram rather thAn report 
it in analytical terms. The aim is to convey something o£ the liftk of c.ha - 
program and the concerns of its inhabit^ts, * ' ^ 

Similarly, Parlett and Hamilton's illuminative evaluation discards tiie 
analytical role in favour of an interpretative one. Parlet and Hajnilton 
claim the authority of social anthropology for their view. On reflection, 
it seems that a wider '^interpretative tradition informs the mGthodology — 
the interpretative tradition of verstghen sociologists (see Outhwaife , 
1975) and the htrmeneutic or interpretative tradition in social "science and 
history (see Kemmis , 1978). The illuminative evaluator chooses issues in a 
situation which help the reader to grasp its wholeness . and to develop an 
empathetic understanding ( verstehen ) of the lives of those who iTihaJaijE it, 

f i " 

These methods, in my view, have much to recommend ' them . They set about 
del-iberately trying to .disclose the life of a social situation, progressivery 
jocussing on^ issues of particular concern' to participants and accumulating^ . 
data along, the lines suggested by those issues. ' " 

The development of the illuminative and responsive models was a response 
to some of the perceived shortcomings of the engineering model. At a 
conference at Churchill College, Cambridge, in late 1972, U.S. and British 
''dissidents" from the engineering ^approach gathered to discuss the '^emergfng 
;alternatives * Among their number were Bpb Stak« (a psychometrician 
'increasingly disenchantfd by the empirical-analytic approaches to evaluation 
which characterised his own former work) , Malcolm Parlett (an experimental ■ 
psychologist who was similarly disenchanted by the limits of analysis and 
had turned to sociology and social anthropology for an alternative), David - 
H^ilton (an ex-geology ^graduate, ex-teacher who had become interested in the 
social processes of classrooms)/ Mike Atkin (one of the prime movers in - 
developing a high--school astronomy course in the U.S. and one of the 
earliest writers bh the shortcomings of the objectives-based approach — 
see Atkin, (1963), Lou Smith (already attracting substantial attention for - 
his ethnographic work in classroom's — see, for example, Smith' and^ 
Geoffrey/ 1968), Barry MacDonald (chief evaluator of the Humanities 
Curriculum Project in Britain , adopting case study approach and 
/concerned ^with the way evaluation could serve decision-'makers and wider ^ 
audiences of evaluation, and becoming sensitive to the politics of 

evaluation) , David Jenkins . (an ex^medieval religious drama scholar., ;eK- 

teacher, evaluator of the Keele Integrated Studies 'Project^ a wit an^ 
raconteur whose brilliance as a writer emergea in his evaluation reporting) , 
and a number of other names now well-known in evaluation circles. 

Towards the end the conference, a ^'manifesto" was draf^ted and endorsed 
by the participants. It was subsei^ently ptiblished in a "reader " ^ on _ . ^ 

alternative evaluation methodologies. Beyond the Nmibers Gone (Haniilton, 
&t al\ , 1977) I ^ ; ^ 

% On 20 Deaembep^ 1078 at Churahill College ^ CmhridgB the aonf^enae^ ^ 

partiQ%pants aonGluded a diBausBion of the airns and proaedures of 
evaluating eduaational pragtiaes and agreed: ' / 

I That pafft efforts to evaluate these praatiaeB have^ on^thm 
- whole^ not adequately served the needs of those who require 

\ evidenae of the effeats of smh praatiaesj beaauim of: . ^ . " 

(a) an under ^attention to eduaational proaesses inaluding 
those of the learning milieu*'^ ' 



(b) an oveT-attrntioruto psyQiwmBtiHoally measia'able ahamjes 
in student behaviowp ( that to an mst&nt represent the = outaomm 
. of the praatiae^ but whioH m*e/^ misleading over^aimplifiQation 
of the Qomplem ahangea that oaaur in studentB) ; and 
. (j) the existenae of an eduaational researah aliwate tJmt 

, rewards ao&u^aGy of measia^&nent ai^d generality of theory hut ^ ^ 
' ^ overlooks both miBmatah betwem sqhooZ problm& and researah ^ 

issues mfd tolerates ineffeative aonmuniaation beUjeeri 
'researGhers and those outside the researah ammunity* * 
II That future efforts to evaluate these praatiaes be designed so 
as to be: 

. . ' (a) iflespon&ive to the needs and perspectives of differing 
. audienaesj - ^ 

(b) ilUminative of the oomplem organisational^ teaahing and 
learning proaedses^at issue j ^ 

(a) relevant to publiq and professional deaisigns forthaoming; ^ 
and ^ „ ^ ' ^ 

(d) reported in \anguage whiah is aaaessible to thei^ audiences 
III More speaifiaally tJiey reaoTrmend that^ inai^easingly ^ 

(a>) observationai data^ o^Sfully yalidated^ be used (sometimes 
= ^ in substitute for data from questionir^ and testing) s 

(b) the evaluation b# designed so as to be flexiblej emugh to ^ 
allow for response to unantiaipated events (progressive foaussii^ 
rather than pre^or^inate, Resign) ; and that 

/ (a) the value positions of the evaluatorj whether highlighted 
or aonstrained by'^the design^ be made evident to the sponsors' 
; and audieiwes of the evaluation, ... I _ 

IV TJwugh without oonsensms on the issues thmselveSj it was c^reed 
that aonsidered attention by those who design valuation atudies 
shauld^be given to suah issues as" the following* - 
(a) the sometimes Gonftiating roles ^f the sme evahmtor as 
empert^ saientist^ guide and teaaher of depision-^makers on the 
one hand^ ard as teahniadt speaialisty mployee aM servant of 
deaision^m.akers on the other j . " = . 

' ^ = . (bj^the degree ^to which the evaluator^ his sponsors and his 

. - subjects, should speaify in advance the .limits of inquiry ^ 

the airaulation of firdings^ and suah matters as may become ; 
=^ Qontroversiat later j ^ , t / " = 

(a), the^ advantages and (disadvantages of intervenir^ in 
' * eduaational practices for tUe ^ purpose of ga:thering data . . , ' 
or of aontrotlii^ the variability of certain feat^es -in 
' order to inarease the "generaUsahility of the find%Y^s;^ 

(d) the complexity of edUGational deaths ions which ^ as a 
matter of mle^ Jm>e pblitiaal^ social and economia , 
mptications; and the responsihility that the evaluator may 

. or may not for emplorir^ these implications; 

(e) the degree to, which the evaluator should interpret his 
observati^s rather tJmn leave thm. for differmt audiences 
to interpret*^ ' ■ - . 

It was aaknowle^ed that different evaluation designs will serve 
... different ppopposes ard that even for a single eduaatiandl 
progrmme many different designs could be used, : 

Looking. back on the "manifasto" from the perspective of 1980/ it i^s hard to 
remember how challenging those .id^as were. They were prefigured by. the work 
a considerable body of students '^of social and educational life^ to be aure^ 
but they represented an alliance of interests mong a rtiomber of disaidents 
from the dominemt evaluation ^paradigm on both sides of the Atlantic - The 
alliance continues to be a yeast to the'" dough of €he evaluation literature . - 



The images reprasented by the illuminative and responsive approaches have 
venerable roots in social science, history *and the 'hi&^anitics . Perhaps it 
is a British characteristic which sLmply lightens the consistency of the 
sometimes extremely tur^gid (and U. S . -dominated) evaluation literature, but 
■ the illuminative^ and responsive approaches ^se^m niDre literate, wittier, and 
m©ra artistic than much of the work reported in the''" ''conventional " tivaluation 
literature,^ Here we see the^ evaluator as social anthrupoiogist , case^study 
worker, historian, portrayai'maker or artist. (Contrast' the role of the 
evaluator as artist with the approach of Eisner who developed the notion of" - 
evaluation ^s art criticism; Eisner, 1974J 

These are^images which not only attract but compel. They attempt to over- 
come the atechnologiaation or reason with an. art form. They attempt to heal 
th& fragmentation of our understanding of a program produced by analytic 
(experimental,^ psychometric , 'objectives^based) methods and to provide 
holistic descriptions on tlie bagis of which, we, as readers, can judge the 
programs they .portray for ourselves. 

But the heal'lng process is not so easily achieved, v ^ ■ • . 

The recipients of the first products df the "new wave" evaluation, aa it ' 
came to be known (Stenhouse, 1975), must have been somewhat surprised by 
the sheer readability of the reports. But the response seemed short-lived: 
Very soon, the reports were being condemned *as ^"subjective" and ■ : 
"impressiQni3tic". ^ It. was. argiied that they .could only be produced by a 
certain' kind of person ;the methods were so, poorly understood that they 
could not be regarded as rigorous by audiences who had come to equate rigour 
with the rigidity of the prevailing! technologS^) , -A long debate ''abbut the 
qualitative vs , the quantitative and thfe objective vs/ subjective in 
evaluation began,, though some might prefer not to describe.it as a 
-/"debate"; opponents ^eemed 'to talk past one another as often as they . 
bumped heads. ' ^ \ ^ . , 

I "have dwelt so long on the topic of illuminative and responsive modefs^ 
because they represent a substantial ctiallenge^ to the engineering model in^ ' ^ 
particular, At^ the challenge has. been 'successful at least to the extent . 
thSt the alternative approaeh has est^lished itself as fegitimate, \ ^ 

But severalv.dif faculties with illuminative and responsive appr®aches ' * ^ 
rem.ain. The first is the problm that it is dismissed as "subjective" and 
"impressionistic". That these responses are based on^ ignorance^ of the 
methodological pedigree of ^the verstehen (empathetia understanding) and ' 
" hermeneutic " . ( interpretative i based on the scholarly tradition and his'tory; 
see Gadamer, 1975) traditions need not concern us here. What should . 
concern us is, that this dismissal of " ill™inative/responsive' evalu^ 
reveals that»'the reports are not, reg.';rded ^s credible/ This is, of^ course, 
a fundamental problem lor an approach which is%uilt on the notibn that 
evaluation should speak to the concerns df participants and in their own 
preferred languages. I take it as an indibatar of the extent 6f the 
technologi^ation of our reason that this failure to treat the reports of 
illumiriative/responsive evaluation seriously should occur ^at W^l* Suqh is 
the imperialism of scientistic thought that it believes there^^are no ^ 
truths to be told outside its own language. (How then dp we justify 
history or criticism? As pastimes for effete intellectuals?) *^ 

The second difficulty is that posed by Walker (1977) in a paper entitled 
"Descriptive methodologies and utilitarian objectives i Is a happy, marriage . 
possible?", (The title almost (Ascribes the whole argument.) The problem 
is that idescriptive^ methodologies, especially whfen they aspire to-acute." .| f 
observatipn and profound insight, .take time . But that • is the commodity 
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' program. eValuators are usually short of. The "utilitarian objectives"^ of 
avaluation — getti ng the Inform ation to the dacision'-maker before t!ie 
.decision to be made = aire .antipathetiG to the dascriptive objectives. 
Walker's argument maty lend support to those who #rgu3 that the reports are 
"subjective" or "impressionistic" if it is read to suggest that they may 
be so because they are^ produced under pressure of time* (I art not 
personariy convinced that this criticism- is fatal to the project of illumi- 
native/responsive evaluation! ^he process of "negotiation of accounts" to _ 
be discussed latar in relation to the "d^ocratic"= approach at least 
mitigates the force of the argmnen^* Moreover/ the process of writing up 
•at least part of the life of a program/ even if imperfectly, may help 
people to make wiser judgements of it; it is my view a crass mistake to 
brieve that no knowledge is Betteif' than arguable knowledge about a 
program,) ' - , 

. A third difficulty is that these approaGhes tehd to fall by one of the ' 
criteria they value most highly: the criterion \of authenticity . The^ 
notions of "responsiveness", to participant and audience concerns , of 
"progressively focussing" on issues which compei those in and aroi^d^the 
situation / and of "participant confirmation" {or\ "the shock of recog^itiori") 
as guarantees of authenticity are Challenged by the notion that dt is not ^ , 
participants ther^selves but profeftsional ev^luators who shape the accounts* 
Perhaps this seems ^a purely technical problem i participiihts don't have the 
time to carry out the evaluation for thraselves, don ' t have t^e appropriate 
skills, can'J; easily mediate ^with evaluation audiencts with whom they have 
ilrMdy'-r^ntermined relationshi But it is not a merely tpchnical considera- 
tion: the outside observer' understanding and reporting on the program is • - 

^doing something fundamentally different from the insider-partiGipant who 
must live with the cohsequences of the evaluation, with the meanings 
reclaimed from its life, and the reputations the evaluation establishes or 
diminishes* . ^ " / . . / - 

l*hough illmninative and respons-iMe ^approaches ^re more-, qonviviaU. than the^r 
forbears in the- evaluatibn literature of the last two- decades , they have 
thisAehilles heeli that the \life of the program is reconstructed for ' 
participants, not b^ them* ^ In this sense, the illmnlTiative/responsive ? 
evaluator still pro^Kides a teGhnical service to the knowledge^'generation 
processes of ^ the 'program cojmunity* By. ppepialising the- knowledge-- 
generation process with a band ot professipnal evalu^tors. Illuminative, and. 
rjesponsive approaches pife^erve a crucial remnant of the technologisation of . 
reason: someone else, not progrm participants or program audiences, knows , 
best about how knowledge of the program is ^ to^ be distilj.ed and reported *3 



^THE D^pCRATIC MODEL - - . ^ - 

The "democratic" approach to program evaluation attempts ^ to provide a 
solution to this problem* It moves from the realm of processes of uhder= 
standing social^ contexts to the question of . the politics of information in 
, evaluating them. In a sense, th6 shift from illmnlnative/responsive to 
democratic models parallels the, shift from the engineering ^to the orgahisa* 
'tibnal model . Both the organisational and the democratic. model incorpbrate 



Note, hdwever, that Stake and s6me co-v)drkers have proposed that program . 
participants develop tKeir own evaluations or portrayals of their own- 
situations* See, for example^ Stake^ et at. The Aaaountability Notebook 
(19 71) Grotelueschen and Kemnls ' (1973)- notion of an evaluatida file,;„^ 
and "stake and, SGheyer 's {1976) self ^evaluation portfolio, " ' , ^ ' 




an elaborated view of the instJ^ti^.onal uses of approach. In a widely- 
cited paper ^ ' "Evaluation and the^»^^is{ducatibn" , MacDonald {1976) 
distinguishes batwaen bureauc c^tfTc ^ \^t6cr^ic and damogratic approachGs 
to the evaluation of prograins.i 

Bureaucratic e^fflTO^iorT' 

Bureauorati^evaluat^on is an unconditlft^l service to those 
goveriwient agencies ^ich ha^ve major contrbl over the ax?ocation 
of educationTBl resourt^s. The ©valuator acce'^^ the values^gf . 
those, who hoi^officei and offers inf orinatwn ^whiGh will hel^ , 
them to acGOmpI^^ taeir policy^ objectives. He a^ts as a mana^^ 

- ment consultant ^ ffl^/his criterion of success is cMent satis^ \^ , 
faction. His techn^^^s of study must' be credible ko .fche pQlicy"-^ 

. makers and not lay |ch^^^ento^g^^^^f iti^«^^He has no 
independence, no c^trpl ove^^^^rtse tha^t is made of his information 
and no court of appeal. The report is owned by the- bureaucracy 
and lodged in its files. The key concepts -of bureaucratic evaluation 
are 'services 'utility' and 'ef-ficiency ' , Its key justificatory,, 
concept is 'the reality of ^power's. ^ ^' 

Amtoc^ratics evalufation ■ / , 

' . Autocratic evaluation is a conditional service to those government 

agencies which have ma jor control over the allocation of* educational 
resources. It offer$ external validation of policy in exchange for 
compliance with its recoimnendations Its 'values are derived from 
the e^aluator's perception of thfe constitutional and moral obligations 
of tfim bureaucracy. He focuses "upon issues of educational merit, and 
acts^ as eKpert advisor. His techniques fef study must yield scientific 
proofs, because his power. base is the academic research 'community , 
Ifis contractual arrang^erits guarantee non-interference by the client, 
^ah4 he retains ownership, of the study. His report is lodged in the 
files of the bureaucracy, but is also published \in academia journals. 
If his. rpcommendations are rejected, policy is npt validated. His, 
Gourt-. of 'appeal is the reseaifch community, and Wigher levels in the 
bureaucrady . : The key concepts of the autocratic evaluator are ^ 
'principle' and 'objectivity ' . its key justificatory concept is 
'.the responsibility ^of office' , 

' ^ DOTiocratic evaluation * . , ' ^ , . ■ 

Democratic evaluatipn is an information service to the coiffiiunity . , 
about the chkrMteristics of an/educatlonal programme. It recognises 
= value-pluralism and seeks " to represfent a range of interests in its 
- - ^ • issUe-rfo^ulationT The basic ah informed citizenry, -and~- 

* - the evaluator acts as broker ia eKchanges of information between 

' differing groups, .His tecbniqWf s of data^gathering and presentation ' 

must be accessible tb non-apecJ^ist audiences , " His main activity 
is the collection o£Vdefinitibnso% and reactions to, the prpgramme. . 
' He offers confidentiality to infomants and gives them^ control, over 

his use of the information. The report is non-recoimendatory, and 
the evaluator has no concept of information misuse. The evaluator 
, engages in periodic^ negotiation of '''hiB relationships with sponsors ; - 
and programme participants. The criterion of success^is the range 
» of ' audiences served. Trie report aspires to 'bestseller ' status , 
The key concepts o4 dSiadratic evaluation are 'confidentiality', 
'negotiation' and 'accessibility'. The ke^ justificatory conbept^* 
im 'the right. to know*. * , * - 

The image of democratic evaluation thus poses a sharp challenge' to the orgmni-- 
eational; model {which is essjentially bureaucratic), .The democratic evaluator 
'stands, outside .the .program >s an^ independent , Ji^interes tad observer though 
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not an "objective" onmt the "objective" outsider is servinf "scientific" 
interests and thus adopts and interested, autocratic: stance. He/she . \ ^ 
attempts to advocate the contending perspectives within it by prox^, fully 
and fairly, to regulate the processes by which participants. come to agree 
that their perspectives are adequately represented (negotiation of accounts) 
and by which these views are gradually released to other' participants in the , 
situatibn (negotiation of release) , The procedures for handling these 
negotiations are now susceptible^-of articulation and forniulation as "principles 
of ^prpcedure"t for evaluation studies (see Kcmmis and Robottom, 1980)* 

The image of the democratic evaluator 'has its racier or more sinister side 
(depending wgon your 'point of vie\^) i it is the image of the investigative 
journalist. This image is largely compatible with the image of illuminative 
or responsive evaluation^ though peehaps it recognises itself as more "reactive" i 
the investigative journalist shapes public affairs: by reporting them; he/she 
-does not marely report, (MacDDnald, 1977, refers eKplicitly to the model of 
the "new journalist" as a source of techniques for the evaluatbr.) But it has 
- . "another •^Jde too i that of the open-minded ^ concerned observer (based on the 

model of the "neutral chaiman" of the Humanities Curriculum Project of which . 
MacDonald was the evaluator; Jenkins^ Kemmis ,and Atkin* 1977) *who attempts to 
see that ^reason, rathei^- than political muscle wins the ^ay as differing views 
= of a program are shared, ^ong participants , and between participants and program 
^ audiences • . • 

; This 1^^ part of the image of. the democratic evaluation = as neutral 
chaiman ii^-a.^_"^ about the program — is especially attractive. 

While in other relpests the d^ocratic approach is subject, to the failings 
of the ill^inative^and responsive, .ft^ it is an attempt to overcome 

\ the political probltoi of authentieTty^--4i^^^ to respept the autonomy \ 

. ' and responsibility and the reasonableness or^MgEam^participants, * ' 

' It. is Clear, however, that the democratic evaluator is still p3foviding_a^ 

technical service to a. program community t that oF^^'neutral chairmanship" in "x^^ 
the'l^debate aboiit the , program , Perha^ thi^s is tdchnologisatian of reasons 
■certainly the techni^es for production and distribution of kiipwledge about 
the prpgriW are procedurally-bound. But I dO; hot think that it can be so 
described: on the contrary / the democratic approach fosters / rather than 
denies, the capacity for critical thinking about the program/ ^ It thus' comes 
■ closest of the ijnages of evaluation so far discussed to achieving both . 
conviviality and critical capacitV*, . ^ - 

Its failing/ if such it may be called, is that the process of democratic 
=:~™.evalua€jd1rr^ is ^^e^of ■ IHCerVention the social 

world of the prograW to pfovide this key service orf behali of program pjrtiGi^ , 
^ > pints, I^iough ^Imay provide^ a model of conv^ivialitjy and 'criticism^ it\^&B 
V not. necessarilyir implant the .mo^^ in the situation /' As the notion is expressed 
, ' ' ny Jurgeh HaberrtOT (1974), the ultijnate prqblm is to create conditions within 
^^^rofr™ through which self -reilect sustaine<^by 
.^rticipants. on their Own behalf*; It is a probl^ of "the organisation of^. 
^ enLightenrtent", guided by an Emancipatory'^ interest (that of rising above 
* 'the dictates of habit, precedent, coercion a:nd material constraints towards 

gysater reasonableness, self -awareness, discip2 tne of thought and self-'control = 
^^and towards greater scl£ -dateanninat ion based on autonomy and responsibility 
and an . awafceness of ',the autonomy and resporisibili^ty of, others). We will 
return to these notionf in a later section. * . 

: . . 1 \^ . '- \.. ' ■ 

THREE GENE^L ISSUES ' ' . ^ ' \. t 

. ' Before proceeding/ I should like to retur^' to three general issues of concern 
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" views of knowledge, the question of conviviality, arid the question of autonomy 
ahd responsibility 1 By returning to these issue's^ I hope to emphasise several 
points made in my suvery of images of program evaluation so far. 

(a) The question of objective arid subjective views of knowledge 

Many avaluators trained in social sciencas ha^/e daveloped-^=peMpaeti-v 
the role and function of soGial science which depends on the notion of . ^ 
"objectivity": the capacity of science to free itself from commitrnents on one 
side or another of a value-'questionv Such avaluators hope that science can 
be value-'frie, - 1 

This position has come under heavy attack in the philosophy of eocial sciences 
in recent years (see^, ^ for just one exmple, Patrie/ 1972). According to these 
writers, soc-ial science is not value-'freef even the qufstions we ask are shaped 
by a frMnework of our own (or our culture's) values and interests* Habermas 
(1972) introduced the concept of ■■knowledge^constitutive" interests ,to 
describe the ways in which our inquiries are shaped to serve our values and 
^oiif xhtmrS&tWV ^ ~ ^~ ~ " ' ■ — ~ — - ^ ^ — _ 



Maturana (1974) outlined ^he difference between the "objective", and "subjectlv^^ 
views in considering the question, of the unity of mankind. According 'to the 
"objective" view^ he suggests, men exist in an objective world that can be 
^ known and which exists independently of us as observers i our sense organs 
givi*' us reli^le knowledge of this world'. The consequence of . this view, he \ 
argues, is the view of mankind which sees us as genetically equivalent 
differing only in levels of andovmient^ in physical intelLpctual terms^v,-^^ 
which sees cultural dj^ferences as /Reflecting different modes of treating - . - 
an objective ^reality I . arid according to which the. pultural unity of man - can 
^nly be obtained through the* development of a culture in which men base - ' : 
thair conduct on objective khowledge that is, through the development 
of a culture " in .which men have "the right way" of looking at 'reality* 

In contradistinction to Jhe /'objective" view, Maturana puts the view of ^ ^ 
cognitibn as. a subject^dependarit phenomenan* ' According to the^ "subjective" 
view, cognition as a process is bound cons€itutively^ to the organisation arid 
structure of the knowe'r; our history of interactions in the world defines^ 
the world for us, and it simultaneously defines us as "unita of Interactions" ^ ■ 
wlth^cirtaih^potdhtials for ^interaction, J^e consequence of this view in 
relation to the 'cultural unity of mankind is that, cultural differaftces do not 
reflect different modes of -treating the mamm objective reality but legi-timately 
different .cognitive domains, ' Cuiturally different men live in different 
cognitive^ realities that are gena^ated and specif ied/^y their living^ in them*, - 
•jnie probrem of the cultural unity of mankind therefore, is not a -problem of 
iearning a single vaJ.id cogjnitive approach to objective reality (as rational 
CTtpiricists would have it) , but the problCT *of generating a coitmon subject- 
leperident behavioural domain. that defines a cpnnnon subj#ct^dependent reality* . 
In^^ort, Matu ma argues that we ^ouM not expegt -to find the cultural ^ J| 
unitys^f man by reference to a coimnon objective reality "out there" ^ as It - 'V| 
were,' bbt by living together so that we' orieht in conmon ways to the #o£ld .'| 
and to one^riot^er. ^ ^ ' ''-a 



ThLm perspectiv^is also implied by Wittgenstein's (1974) notions of "forms 
of life"' and "lang^aage games"* Truth, it may Be argued, is not to be found* 
entirely within langmage (empirical are about states^of^af fairs in the 
world) nor is it. to bfe^fdund entirely, outside language (it relates to 
prppositiona, not to eKp^^ancea alone) * Wittgenstein piits the problem tthus: 



, ' -So you mm saying tJmi humn agremmt d&fims what is true and - 

wMt is false? — It is ikhat^h'wmn beings say that is true afid 
■ fats e; and they ^jagree in the language they u6e. That is Hot 
a^rSWient in opinions hut in form of life* (1974^ p. 88) 

/ ^ ^ - -i' ^ ' 

Th^ solution" to the problem "how de decide if a thing is true?" depends 
u|K?n the way we use iangTOg e \ ' wh ic h ih' tuYri ^epeftdB ~ u porr^o^r 'Itay ing^ ^ "foCTs; ' - 
of life and sufficiently comprahensible to ,one another to support , 
Cdiranunication * : * ^ . = ^ , . 

ay discussing the question .of the 'bbjectlve" and the "subjectiva" . in ''relation 
to the ideas of Maturana and Wittgenstein^ I have hoped to show that the 
"subjective"/ the bane of "objective" social science, is not m^e opinion, 
not merfe individualism^ of perspective. Both tha "objactiva" and tha 
"subjective" depend upon agreemehts about how language is to be used. We 
have in fact, arrived at an "interactive^subjective" perspective (not the 
lone subject experiencing and^ yet un^le to test or to communicata that 
eKperiene-e) . ^ ^ t . * 

The 'images of progr^ evaluation discussed so far rfiay be arranged along this ^ 
dimension from objective tO' subjective. The "objectiva" approaches under- 
estimate^ the manifold quality of the world we experience, and they fragifeaAt 
our understanding , of it. The' most "subjective" imaga presented here 
(illuminative/riasponsive) is not yet fully conscious of^its potential 
conviviality — its qapacity to entar ' the program being studied, *not to 
stand aloof from it as observer. _ - . ^ 

Many, social scientists* hope that the' battle "between the "objective^" and tha 
*^siabj active" can^be wan at that level* But some r^ent philosophers of 
sdciil "scienca do not^think thate this can happen,. For example, von Wright ' 
in his book EmptoTrntiori'^cmd Understa^ir^ il97lj , argues that tha two are ; 
simply incommensurable, that they are different modes which can never be. 
reconciled I HabeCTias/ in Theory cmd Praatiae (1974), however, argues that^ 
both are limited but they can be transcanded in critical seli-ref lectio?! • V 
Choose your own .unhappy endi^igp . ■ , 

This brief excursion into, the world 'of the.."pbjective" and: "subjacti^ 
may ■convince you of litttle. - But it, tnay at 'least have raised the possibility ~ 
that, to be. convit/iai. and tb engage ioui^^ under standing, program avaluation must 
have/at least some iiement of the "subjective". Without the human process of 
expressing our eHP^^ienee in coMnunication ^nd orienting to one another in 
coimnon- action, we cannot hope fully 'to und^stand. This Lb dQwnfall of 
tha "objective" evaluator, as it is of th^ a valuator , who stands on the- 
sidelines of the actior* as comnantator or critic. As Mao Tastung once_ put 

'^If you want :to underatand^ reality ^ you must pwptioipate in the struggle' 
to aJmnge it-^> ^' . ^ ■ ^ .'f' 



ih) Conviviality 

• 4 
Sarry Wolcott , i 

paper; ^i^en at the Annual Meeting of \ the ^ericain Association for Research in 



Harry Wolcott , an^ ethnographer pf education, was called upon to discuss a 



Wolc^tt's (1977) book Teaohers vi\ Teahnoarats bears' greatly oq our/ 
.discussion — being both an ethnography (and thus a potentiai source of 

justification for advocates of the illuminative model of evaluation) and 

a treatment lOf the problTOi of rack o£ conviviality in program ntanagement. 

and evaluation using , the technology of PPBS ^ffrogran Plarthing' and ' • 
:^^Bud€fetina->Svsteml.:.. . , . ^.' ' ^\ / = ■ - . ■ ■ n =v..:.- . y, v-i,>- 
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EdiTcation, in San Francisco in li76 by two evaluators of a technical/ 
engineering/organisational cast of mind. These svaluators had defined ^ 
education as "the collection of data- to guide decisions to continue^ rev^ise 
or terminate programs"* '"J/ you w&re an et?imgrapher^% Wolcott remarked^/ 
"haw would you €ike yoiw material to b& u^ed to, aont'inue^ i^evi&e or 
t&Tminate another aulturef 



'The question is a pressing one for program eyaluatorsi by adopting a 
technical role with respact to program participants the evaluator 
necessa^iiy'P^ts himself outside the** life world cte participants. Vet they 
must live with the consequences of the "truths'* the evaluator tells. 

In a recent paper on case study in research and evaluation (Kemmis^ 1979) 
1 argued that evaluators are ^ or seem to be , ruled by three "tyrannies"; 

= ' ft ' ' ' " 

. - . (1) the tyranny .th&r& is "a truth" to be told ar^ that w& aan 

^ . tali itj / 

'( % ^ (2) the tyranny of aonaaienoe: that we believe Jiave a duty to. 

arrive at Qudg0nents of the ^worth of Boaial and ^eduaational 
' progrcw7&j ar^ tQ aommnioate theB& judgments = our JudggmentB — ^ 
to oth^r&^^and ' , " 

(3) the tyranny of reporting : that we have a duty to aormit these 
"truvha" and' judgment B to paper and ^ through therris to areate 
"the publia meaning" (or at leaBt "a publia meanii^") of the 
work. , ^ • V 

' 1 see these tyrannies as evidence' of the teQhnologiation of ou^ reason /as ^ 

evalua.tors ^ , as evidence of what Sartre (1957^ would describe as "bad^faith" * 
our willinghess to subjugate ourselves to a role thrust upon us so^ietimes , 

? tty eifctinvstahce and ^omitijaes by sponsors offering reputation or financial ; 
' reward* -(i am reminded of a Goori Show in which Eccles had been left to . 
guard the door-^hd keep out newspaper Reporters while^ a secret meet i*ng.4;oQj^^ ^ 
place. 'The reporters , managed to get itito the meeting and Neddy' *Seagoon was 
draandirig an explanation from Ecdles.. His reply? "They burst through by ^ 
fording money into my ?umd"^) l ^ ^ . ^ ' - 

TTie technologi^ation of out reason as evaluators and its attendant bad faith , . 
' is made possible by the perspective we are irivited to take on the /programs . > 
we . are observers , program participants the observed i we are t he specialists T - • 
" the progrMi contains pathologies we are ^ to diagnose i we are the therapists/ ^-^ 
^ participants miist be^ dem^tif iedi we are the disinterested enquirers ^ 

participants are bouod by self-deception and self-iiiterest that only we can 

unravel and. dispel T It is riot surprising jtiiat we submit to the bad faith ^ / 

of the role I we are paid well for our work in money/ opportunities or ^ ' 

reputation. And^it is we, after all/ who/ are ^the reputation-makers / with 

the power to distribute praise or blame/ to ponfer the appellations 

"f^nure"/ "modest failure"/ "noble failure"/ "success". , ^ : 



1 ^ 



Perhaps this characterisation s^ots, extra^^agant or glib*- But it is 
^oser to the truth of the program evaluator 's rolf than many would ewe 
or dare to admit, ^t is certainly the way hn which we are "pertfeived a* 
gr^at number of tHose who siif fervour se^iaes* And it is often the role we' 
are enjoined to accept (quite' deiiberately) by evaluation "Sponsors — we can 
say harsh things and go away/ the* sponsors (who are very often the sponsors ' 
or administrators of the program too) must stay to ^live and rule * another 
day. \ . ' • - -= \ ^ ■-. ' ■ ' ^ 

Earii&r> 1 jpointi^ out that tlie value of authantic ^knowledge is that it. pr^o- 
::viaei'^.:basfa:V ' 
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ticipants* As lllich (1971), Freire (1970) and Habarmas (1974) have been at - 
pains to show, the cq^pacity for c»itical sfelf-r©f lection can only be developed 
in the Iknguage and perspectives of. participants if *the phanges in a program - 
are t^ "take" by conviction rather than coercion, Aj prpgram evaiuators.^out^ 
side other people's progranis, wa cannot tell* onLyi our truths i we must 
'pa'rtieipate in the process of developing critical self-reflection alongsida 
participants, al^'^participants * ^ * * ^. - - 

Thi0 is the way ^of conviviality. It is the ^development of critical self^ - 
reflection ojut of the talk and the (authentic) experience of co^ • 
participants. Langi^age creates the .possibility of critical thought 
(though it, never guarantees It),, The language of critical' thought must ^ 
recognise the experience and the language of those who want to participate 
in the critical debute I It must be self - reflection as well as critique of 
tha social world of the progrMip , " 

(c) Autonomy and res^nsibility 

In any discussion of program evaluation, the question of autonomy and 
responsibility fnust arise* It is^ inextricably linked to the* question of 
contrdl of the program^ since evaluation ^ whenever it aspires tO' 
influenpe progrM action/ concerns program control, ' ' ^ ' . 

Moral responsibility for an outcome can only be ascribed to a person to th^ 
degree that his or her free choice of actipn &b an autonomous moral agent, 
was a cause of that outcome. Just as th6 evaluator must' assise that th^S6 
involved in a progrgupa^'a^ rationally in^he sinie^hat^ they .art open to ; ^ v ^_ 
arguments based on reason/ so it must be ass Wed that they will act as ^ V. 
autonomous' and responsible moral agents* _ 



This said, it becomes evident that a progim is a cotnmunity of self -interests 
" , (House' and Gare/ 1977) based oh ^the comm^-tatients of - par tic-ipants within the , 
limits of the^ir freedom, the *opportunitieB and cirgtimstances of the program 
^""^^^tin f and ^thi^ knowledge available about the consequences C intended and^ 
^ uninte ided, anticipated and unanticipated) of the progamm-;^ ^ ' 

' The au nonon^ and responsibility of those invoLved in a program 'i^ misianed, by 
somi*Q| the images of program ev^^atidn discussed here and ^negated by others * 
Indeed^ ^ it is a feature of the more "technological" models (especially* the 
•engineering model and the organisational model) that p^rtricipants are treated 

v,as having subjugated themselves to the techijology of the progrto* (In thp 
use of the ecological model , it would appear. that actors are assufned to.be, s 
L to a ' greater, or lesser, degree of systemic constraints, dete«nined rather than 
free to act. as moral agents J Program, conditions which.^ require this 

^ subjugation. of participants^ may be coercive; program evaluators who 
assume this subjugation of participants are likewise coercive 

. . 9 _ ' ' . / \ \ . ' ^ ^ \ ... ' . , 

These three questibns of ."subjective" and "objective" views of knowledge, ^ 
conviviality # aM autohomy'and responsibility must be actively Considered 
in the development of a model of program evaluation capable of avbidlhg the , 
limitations of the images ^bf evaluation presented earlier, it 'is with th^se 

/ questiops in mind that I now propose^ a sixth model of program evaluation 



evaluation as seiy-r^fflection ,in^ U^e coimnunity of self-interests formed \byv - 
a prograin* / • , « ^ ^ • " ^ r— ~^ 1 

mfALUATlON AS SE^-RfiPLECTION ^ IN A- CRITICAL COMMUNITY ' Q /I ^ 

Jevons (1979) -has reinarked that the strategic aim of a tertiary, education ^| 
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has also raised the 'genesal , question "what is organisation that it may 
learn?" He demonstrates that organieations do indeed leatn from "their , 

experience. ' ^ 

^ * $ 

Habermas^. (1974p^hms shown how this aim^Gan be achieved in social drgar^iga- 
tionp 'by the creation of conditions for "the organisation of enlightenment' 
He points out that three kinds of questions, each with its own criteria' of 
adequacy^ must b# addressed^.^in the organisation of enligh tenment and the 
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justification of action* First, at the level of s c 1 e n 1 1 f 1 c d Lsc ou r s e / 
critical i&aas which build towards a theory of the organisation must be 
developed. = The^e must be teste^ against the usual canons of truth-telling* 
-Second^ at the level oS the orgariisati^n of j anlighterment itself-, processes 
must be established which allow participants to- engage in critlc^^ self- ^ 
-'refiection on their own thepretical perspectives and practlcmi conmiitments* 
These are to be judged by*the criterion ;of fred conunitment t^ self-- . 
reflfection and to the critical,! co-operativ'e project of establishing a ' 
consensus about social .action based on mutual, understanding and recognition 
, of participants as persons in their striving for consensus* Thirds at the. 
- " level " of. . practical action , processes must be established according to which 
practice can be improved and better ^dafted^ to the' available opportunities 
^ and ci^cmnstances in/,the light of participants'^ intentions Practical 4 , 
* action must be . judged by reference to the criteria' of the ^wisdom and 
prudence of decisions about strategic action*. 

These three "let^els" provide the substance, for critical self -reflection. 
Kemsais and Hughes (1979) discussed evaluation as sePf-rtflection in a critical 
;* . conOTunity using these 'levels as substance for the discussions/ and outlined 
ah example of the = approach in practice* . . 

Beyond thii substance , there is a aeed to^Qreate.a climate in which seltj 
, . reflective debate can be carried ayit. ^ The appropriate climate as one ; = 

described by the .criterion for the organisation of enlightenments that 
, ' debatp. be- carried.^ out- in ar> atmosphere of respeqt for persons and' striving g 

for consensus and mutual, understanding* Kenmiis (1980a>. describes a.stEategy^ " 
; for .developing °the skills df "syimneltrical communications*' nBcessary to ' . 

establish such a climate *= - ' - - V ,^ 

^Sayen principles for progr^ evaluation describe the*^ approach. advocated 
.heSe (Kernmis, 1980b)^ Tdgether.^_ they create an -^age of evaluation as the - 
,* process d£ marshalling 'iofomation and argmnents which, enables in'terested 
^ indrviduais^ and groups to partipipatis in the critical debateMthe ^maBms, 
• pf self -re flection r ^out a program . j ' -. ^ ^ ' .. ' ^ 

These principles a.f e : . ' ' . * ' ' ^ ^ 

' ' (D ^ ^tlte prina^pim .of rationality as reaBonabl&n&aB ; \ ^ 

ppogfrmp pa^tiaipantB aat Vma$on&bly^n the %i^t > of their ^ v v 
airmff^B^T^ea ai^ oppprUinitieB^ It ia the ta$k of^a^ ' - . 
" " \ mvatiiat%on- to illt4mifmte tfi.e reas^ whiph guide$ progpom 

' . ' d&^elopment and wolution^ to identify ^the^:a6ntmUi^ and . 
* hiBtoriaal faGtdTB*whiQh influ&r^B it^^ and to^faqilitate * 

. ' aritiaal emam^iaiion of these matters in ar^ around the ^ ^ - ' 

, ....... ^ . *» ■ ^ . . . 

progrmt aommmtty, ' . - , " : - . : 

' (8) Thk' prinaipie of mtommy ^cmd *rmBponBibility - ^ . ' • / J 
^2^ J7^#apOTaiMZi% /or -on outbdme^aan onty he aaQrib^d to ^ 
■ ' ^ « ' ■ a peTBon to the degree that^Ms or he^ fi^ee ^^iae "of .aation 

. - ' ' = us^ ^m% €mtonomou& 
O " ^:J'^ . O^' 'Oh^icn^lm' d0Oelozment progrmiB are gp-dper.qtipe^^enterpri8s& 




^#al fabtors limit indiviaual8 ^'--^aountabiii foi^ progrmn 
, . actions and their GOns&qu0mjm^ Eualuator§ rnuBt itiurriinate,^ 
'the interaative qhs^aater of (zaQpuntahiiity for a pnogvam. 

(3) Vhe^ Winaipl& of adrrm^nitti'B&M-intemsts ' 

l/hen a &M^T%ffulwh deveto^ent nrojeat i& fomied^ it^ia a 
Qomunity of Belf^int^eresis ^it represente the s&if- 
interests of all pcLPtiaipan-^within its temis of^refeTjehae^ 
The evaluator has a resp6n&4bi^ity tO' illwninat^ the emtent' 
of^vormonaliiy a^^unfliat mnong the values and intermta 
* of p^tic^ipants in this Qomrmnity^ 



m (4) The prinaiplf of pluralttij 



of value^per&peotiveB 



(5) 




'perspeaiiU48 beGom&B ^^&levqnt 
uatvon should identify th&Bs 
responsive to the different 



A rafw0 of different valu 
' jiidg^mg a progrmi^ Ar^ e^" 
differant perspeotives ^ 
Qoi^ems they imply 

The^pmnaiple of the self^G^timt agrmunityi intermt 
evaluS ^ionj evaluation GonsuVicmay ^ meta'^^.valuation^ ^t&rnal 



and independent efiatuafion 
Criiiaal debate about ^e na 
exisiiB wiikhin and c^ound it^ 
tasTt 0Lf prpgram evaluatidn to 



m and worth of a prog^om already 
ogr'dfi aorrmun^y. It is the 
efine this ddbate ahd inprove 
'if8 beb^-0igpn program -aatidn^' ^ytaluation ion&ultanay may ^ 
provide additional tools folp this^pmpdm^ Meta^kvaluation • 
'efforts may help to improve the quality of the aontribution 
of d.progrm ev^Uiation, An emterml evalwLtion may aontri- 
bute to the aritioal debate' by . inGrm&ing cnj)a2*eneBe of a „ 
partiQular set vf values' and intereAta relevant to^ program; 
^t should not be ^thought of as an alperr^Hve 'to the self- ' , 
aritiaal proaess, /.An .p^ependent evali^Won help to. 
tmnress progrm self^aritiGim where the progrm Mmmunity 
/is diffuse or divided by aontiFoversy . Self-critiaism by the 
progrm aormnunity is th^ primary basis [for progrm ' evaluation; 
, other evaluation efforts dctend^it in different ways but do ^ . 

not supplant it, . * . . 

. - - \ - V. . 

($) The prinaiple of propriety in tJie firoduation and distribution 
' of information ■ ' ^ , - ' , . 

Evaluation proaes&es inev.itaMy affect, the politiaal eaor^J^y of_ 
information in a progrmi' (the pro^a'tion and distribution of ^ 
inforrnation about it). Bemuse inf oration and m^gummto ■ 
justify or legitimise deaisions^ evaluation affects ■ the - 
distribution of poWer and resources in program, sitmtidj^, 
Bt'ogrm pc^tiaipants and interested' obserw^em tiv^ with the ^ 
aoneequm^es of use and abuse ^f evalu&tpbn information. An = 
evaUiation should have mpUQitprimipieBofprbQedure whiah 
govern kts aonduat aii^ its proaesses of information praduation 
ahd diBLtribution. ^ . _ . 

(7) The pHnSiple of appropriateness c ^ " / ' ^ 

' Evaluation design is a praQtiaat matter*- An evaluation must be 
. ccpproprwLte to the progrmn setting^ responsible to progr^- 
issue$^ anH relevant to the prbgrm aommn^y and interested 
observer^u' An evaluation design mist be renegotiated as the 
,.. 8^dy progresses- in the lighf of a}{dng%r^ &iramBtance§y 
' fi&Bm^^oM interests s and in the light df its bun gons^quenaes^ >^ 
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This, then is the sixth and final image of program evaiuation t.n bo 
presented here. As distance educators, we must create the mewiiis to 
engage the community of participants in our programs (administraturs , 
teachers, students and others) in the critical d^ate about tlium . Mure 
than this, we must refine the critical debate accordinq to the principles 
outlined here. In short, we must make the community of the- proijrcijri u 
coimnunity of inquirers i a group jointly cdmmitted to develoi -inq' thr program 
^ as an. eKpression of our common striving ^r mutual understanding and 
consensus about social action* 



Some methods for pursuing this goal are available within the evaluation 
literature. -Others are being developed in wider contexts^ one is the 
5 notion of "action research" (see, for exOTple, Corey, 1953) which T .im 
cprrently investigating in a rt search project (funded by ERDC) with Robin 
McTaggart (Kenunis, 1980c)* We are exploring the potential of these methods ^ 
for creating critical communities of course developers, between teachers and 
students, and between schools and their conununities . (Other collhbor itive - 
research is being carried out in these areas at Deakin through the work of 
. the Deakin University Action ^Re'search Group.) ^ 

. The self -reflective approach is in some senses ^he most familiar notion of 
program evaluation* It Is evident in a latent form whenover program 
participants discuss the life' of. a program with a view to understanding it 
in general, with a view to understanding it within the constraints of its 
context and circumstances (perhaps attempting to change its circumstances 
in Order to overcome some of its rimitat ions) , or with a view to modifying 
program practice. In such circumstances, participants engage "naturally" 
in critical self --ref lection on the principle of "concrete negation"/ that 
is, att^pting to supersede th^ real constraints of present circumstances 
^ or present practice in ttie light of wider . considerations * 

Keiranis and Hughes fl979) proposed that this "naturally-^occuring" form of 
self -reflection could be disciplined and extended by^ organisation of 
meetings of program participants or the primary purpose of critical 
self -re/lectibn , ^s ,a conscious attempt to orchestrate the powers of 
critique in the program and tKus to contribute to its theory, organisation 
and pradtice* Su^h meetings. simultaneousLy contribute to the 
improvement of justification of the program and program action ' 

Self =^ref lective meetings provide a focus- and Jorum for critical ^debate 
about the program (a basis" for prQgram critique) • They dp not supplant 
critical self =ref lection elsewhere in the prograirti rather, th^y proinnte 
i,fcs general -.davelopment * Moreover, self ^reflective meetings can 
' involve a range Of different groups relevant to 'the ^programi teachers; ^ 
' students, adipinistrators, sponsors, clients and intereste^d observers 

can be invited to. attend. In this way (cautiously and progressively) these 
groups can make their perspectives known and have them incorporated in (or 
excluded frqm) program justif ioations and action, \ - . 

. = The meetings may have °a slightly ritualistic quality in the sense that' 
t!\ey do^rovide this forum and 'focus , A touch of formality may need tb 
be present inl the proceedings (a. chairperson, 'eicplicit rules of order) 
in order that disparate views can be expressed' and s-eonsideMd adequately * 
; '\ Nevertheless, the procedtSral Aim must be tb eitabJti^ symmetr^al 

. cpmm^^cations as a Way of working towards ^^utukl. understanding and ■ ' /. . 
consensus., Th\is* part^of the procedure' must be to' solicit.'^ perspectives 
^ ^ ■ • acifoss levels of the .hierarchy of. program organisation and administration ^ • 
(if It is = hiararchicaily organised, as most "program^'* are) , and across 
^degyrees XDf. participation^ (from "Insi^der"" to "outsider"). Naturally, ttfe 
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meeting itself must have powers of decision which are binding on 
participants if it. is not to be abused or co-opted by administrators as 
a means for pseudo-'rational coercion or domination of participants (that 
is, subjugating the critical process to administrative self-interests). 

The general problem with which this paper began.- was the cultural problem 
of the technologisatidn of reason. This technologisation undermines our 
capacity to engage in critical self-ref lection . It is most evident in the 
engineering arid organisational modals for program evaluation, but the 
ecoloqical, illuminative/responsive and democratic models are not yet 
complete in overthrqwing its shackles. In one way or another, they limit 
^uthen^ic^ critique. At the general Iftvel , the technologisation of reason 
undermines our capacity to take a critical view of our own culture; at 
the particular level, it undermines our capacity to take a crxtical vxew 
of the educational programs we develop and of which we are member^% The 
methdd of self-ref lection in a critical community is an explicit attempt 
to create the conditions' for authentic critique. , 
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